Chapter 11 XML
This chapter shows you how to process the recently released BNC 2014, which is by far the largest representative collection of spoken English collected in UK. For the purpose of our in-class tutorials, I have included a small sample of the BNC2014 in our demo_data. However, the whole dataset is now available via the official website: British National Corpus 2014. Please sign up for the complete access to the corpus if you need this corpus for your own research.
11.1 BNC Spoken 2014
XML is similar to HTML. Before you process the data, you need to understand the structure of the XML tags in the files. Other than that, the steps are pretty much similar to what we have done before.
First, we read the XML using read_html():
Now it is intuitive that our next step is to extract all utterances (with the tag of <u>...</u>) in the XML file. So you may want to do the following:
## [1] "\r\nanhourlaterhopeshestaysdownratherlate"
## [2] "\r\nwellshehadthosetwohoursearlier"
## [3] "\r\nyeahIknowbutthat'swhywe'reanhourlateisn'tit?mmI'mtirednow"
## [4] "\r\n"
## [5] "\r\ndidyoutext--ANONnameM"
## [6] "\r\nyeahyeahhewrotebacknobotherlad"See the problem?
Using the above method, you lose the word boundary information from the corpus.
What if you do the following?
## [1] "an" "hour" "later" "hope" "she" "stays" "down"
## [8] "rather" "late" "well" "she" "had" "those" "two"
## [15] "hours" "earlier" "yeah" "I" "know" "but"At the first sight, probably it seems that we have solved the problem but we don’t. There are even more problems created:
- Our second method does not extract non-word tokens within each utterance (e.g.,
<pause .../>,<vocal .../>) - Our second method loses the utterance information (i.e., we don’t know which utterance each word belongs to)
Exercise 11.1 Please come up with a way to extract both words and non-word tokens from each utterance. Ideally, the resulting data frame would consist of rows being the utterances, and columns recording the attributes of each autterances.
Most importantly, the data frame should record not only the tokens of the utterance but at the same time the token-level attributes of each word/non-word token as well, e.g., the parts-of-speech, duration of pause etc.# xml_to_df function
read_xml_bnc2014 <- function(xmlfile){
# read one file at a time
corp_bnc<-read_html(xmlfile)
# word/pause token nodes
node_w <- corp_bnc %>% html_nodes(xpath="(//w)|(//pause)")
# u nodes
node_u <- corp_bnc %>% html_nodes(xpath="//u")
node_u_id <- node_u %>% html_attr(name = "n")
node_u_who <- node_u %>% html_attr(name ="who")
node_u_trans <- node_u %>% html_attr(name="trans")
node_u_whoConfidence <- node_u %>% html_attr(name="whoConfidence")
# Define function
# to extract word-based data frame from each u nodeset
extractWords <- function(u_node){
cur_w_nodeset <- u_node %>% html_children()
corp_df <- tibble(
word = cur_w_nodeset %>% html_text,
lemma = cur_w_nodeset %>% html_attr(name="lemma", default="@"),
pos = cur_w_nodeset %>% html_attr(name="pos", default="@"),
usas = cur_w_nodeset %>% html_attr(name = "usas", default="@"),
dur = cur_w_nodeset %>% html_attr(name = "dur", default="@"))
return(corp_df %>%
mutate(word = ifelse(word=="","<PAUSE>",word)))
}# endfunc
extractWords(node_u[[1]]) ->x
x
x$word[4]==""
# create tidy word-based df for current xml file
corp_df <- tibble(
node_u_id,
node_u_who,
node_u_trans,
node_w = map(node_u, extractWords)
) %>%
unnest_tokens(word, node_w, token = function(x) map(x, function(y)
str_c(y$word, y$lemma, y$pos, y$usas, y$dur, sep="_"))) %>%
mutate(xmlid = basename(xmlfile)) %>%
separate(word, into = c("word","lemma","pos","usas","dur"), sep="_")
corp_df
} # endfunc11.2 Process the Whole Directory of BNC2014 Sample
11.2.1 Define Function
In Section 11.1, if you have figured how to extract utterances as well as token-based information from the xml file, you can easily wrap the whole procedure as one function. With this function, we can perform the same procedure to all the xml files of the BNC2014.
For example, let’s assume that we have defined a function:
read_xml_bnc2014 <- function(xml){
...
}
This function takes one xml file as an argument and return a data frame, consisting of utterances and other relevant information from the xml.
Exercise 11.2 Now your job is to write this function, read_xml_BNC2014().
11.2.2 Process the all files in the Directory
Now we utilize the self-defined function, read_xml_BNC2014(), and process all xml files in the demo_data/corp-bnc-spoken2014-sample/. Also, we combine the data.frame returned from each xml into a bigger one, i.e., corp_bnc_df:
s.t <- Sys.time()
bnc_flist <- dir("demo_data/corp-bnc-spoken2014-sample/",full.names = T)
corp_bnc_df <- map(bnc_flist, function(x) read_xml_bnc2014(x)) %>%
do.call(rbind, .)
Sys.time()-s.t## Time difference of 2.161701 minsIt takes about one and half minute to process the sample directory. You may store this corp_bnc_df data frame output for later use so that you don’t have to process the XML files every time you work on BNC2014.
11.3 Metadata
The best thing about BNC2014 is its rich demographic information relating to the settings and speakers of the conversations collected. The whole corpus comes with two metadata sets:
bnc2014spoken-textdata.tsv: metadata for each text transcriptbnc2014spoken-speakerdata.tsv: metadata for each speaker ID
These two metadata sets allow us to get more information about each transcript as well as the speakers in those transcripts.
11.3.1 Text Metadata
bnc_text_meta <- read_tsv("demo_data/corp-bnc-spoken2014-metadata/bnc2014spoken-textdata.tsv", col_names = FALSE)
bnc_text_meta_names <-read_tsv("demo_data/corp-bnc-spoken2014-metadata/metadata-fields-text.txt", skip =2, col_names = F)
names(bnc_text_meta) <- c("textid", bnc_text_meta_names$X2)
bnc_text_meta11.3.2 Speaker Metadata
bnc_sp_meta <- read_tsv("demo_data/corp-bnc-spoken2014-metadata/bnc2014spoken-speakerdata.tsv", col_names = F)
bnc_sp_meta_names <- read_tsv("demo_data/corp-bnc-spoken2014-metadata/metadata-fields-speaker.txt", skip = 3, col_names = F)
names(bnc_sp_meta) <- c("spid", bnc_sp_meta_names$X2)
bnc_sp_meta